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ABSIBACT 

Tha judgment of weather or not a city's Title I 
reading program is a success is dependant upon the length of time 
used in the evaluation* This study's findings that estimates of 
achievement rates and judgments of success were dramatically affected 
by tha length of iima used in the evAluation ¥aa consistent across 
ail the^compenaatory education programs investigated* Specifically, 
tha inclusion of the summer months in an evaluation often reduces 
conaidarahly estimates of achiavemant and hence monthly achievement 
rates. In other words, achievement gains made during the school year 
were not sustained even until the ne^t fali:* These findings are based 
on analysis of matched, longitudinal aamplas containing over 8,700 
students and are unaffected by grade level, the subject area, the 
program, the standardiEad achievement test or the metric used in the 
analysis. (EC) 
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SEI'i study of thi ichiiviiitiit itodinti in ISIA litli I prQ|riii 
bigin M i ripdft miM "Pattarni in Eii Tltli I Eiadia| kUiViMnt." 
li raport uisd as its pfiiary data ioieci lix ygars of state Tltla I 
iviluitioQ uptu iuppliigntid I7 iatirviii^a with mim iiiberi dI itate 
Tltli rofficesi Sisid on the inilpii d these datij Title I sgeiid to 
bi hiving a lubstintial mpaet on the riadicil ichiivaiint of diiadvintipd 
studenti, For lart pida and laeh jgar tm Wl to 19J4j the ivirapi of 
thi states' sonthlj pins weri coniiitently mn 1.1 padi-iquivilint sionthi,. 
Since tteii rates eKceeded the linoffidal standard of luceiss, AiA li an 
avirigi lonthly gain of exactly one gridi-iquivalint aonthj litli I ippearid 
to be quite luecessful. 

lo yirify thesa coseiuilonsj ihf riport thin inilyiidjhe riialti of 
thrii ititiwldi annuil tiitin| progriiBi* If Title I win hivin| 1 sub- 
stantial iipact on achimenti, m ixpeeted to detect an ypi^ard ahift over 
tiii In tk mtu of low^pirceiitile itudenti (those studante lost likely 
to be in litli I propias) . However} since the inciptisn of Title I3 



there his biin no 



shift* In fact, m to not able to ditiet 



any litla I iapaet on aehlewnt, . . = 

map^cted. that the, ippiririt contradictisR in our flndlnp mi caiieed 
by thi ittcriaiei, in achievMnt not being luitainad durln| the sumer Hhlch ' 
followed the prgpn,' k Mjor goil of Jitle I ii to incriaie the expected 
achiivenient of diiidvinti|id studints in order, to ifflprovi thiir icadiilc ... 
futures. But to ^hingi itudinti' acadiffllc futuriij the Increiie in ichlevi^ 
%mt fflust peraiitj |t Um over the smer foUowini a schoo^year progf aa, 



To evaluate whithir or not gains mu being luitainid and to rtiolvi 
thi paradoxical results of our first report, SH conductid ftro furthir 
atudiei of the achievesint of cQspiniatory education studentSp osi study 
for the Office of the 4ssiitant Eieritarf for Idiication and oni for the 
lationsl Institute of Educitlon, Thiii itudiii inilyiid data Im evalu* 
atioiis of Biveral cofflpeasitory education propuSi All the pro|raii 
spanned saveral grades and had coUicted data annually in both tbe^ fall ■ ■ 
and apring, The data froi each pro|ris ofiginiUy hid bun eollictid ai 
part gf an annual evalmtion and, thiiiforej ware cfoss-iictional by yeari 
By ffiitcbing itudents across tm and teat idiiniitratiodij m mn Ah 
to create longitudinal data that pimitted us to deteminej for a given 
pQup of students, the extent to which achiiViiint |aini wiri suetainidj ■ 
It liait until thi beginning of the fiiixt idiooi, 

lli present in this papir one let of data that illustratee our find* 
ingi and a amary of all the prograas itudied, .Je obtilned data frgi. . 
students who hid partieipitid in 1 city-widi TiUa I riiding prograi in 
a Ufp iiidwestirn city, Mtl m ^ill refer to as City M, our 
Mlyses wera perfoied in grade^equivalints since we could only obtain . 
pdi-equivalent scorei, For each gride levil, a supli wis def Inid 
coniisting only of itudenti who raceivid the Seadin| C^prehension section 
of the Gates-MacGinitie Reiding.Ieit three conieeutivi tisss! fall and 
ipring of one school yiar and fall of the next school yiir* To ivaluiti 
the extent to which achiiveiint gaini wifi iuitainedj we calculited thi 
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sample mearis for the three teat administrations and compared rates of 
achievement from two different periods of time* the traditional fall«to« 
spring evaluation period and a IZ-^monthj fall-to*fal4 period. 

The first three oolimns of Tahle 1 contain the means and sta.ndard 
deviations for these iampleg. Since we used data from a matched longitu- 
dinal sample 5 all means for each grade are based on exactly the same group 
of students. An eKaminatlon of the means shows that students make large 
gains during the school year and suffer large losses over the summer. We 
converted the gains into monthly .rates of achievement for two time periods ^ 
fall to spring and fall to fall, by dividing each gain by the number of 
months between the two test administrations (see table footnote) . If the 
program were judged on the basis of only the fall and spring means, the 
program would be considered a success'^-^gains and rates of achievement are 
higher than eKpected* For exMple^ the fall-to^spring rates of achieve- 
ment in Column IV are consistently greater than the month* for^month stand- 
ard for success p However, if the program were judged on the basin of 
the two fall means**that is, judged over a 12-month period-*the program 
appears to have little tapact. The rates of achievement in Column V are 
about what would be expected without any compensatory education program. 
Figure 1 presents the same results graphlcallys 

Clearly, the jud^ent of whether or not City M's reading program is 
a success is dependent upon the length of time used in the evaluation. 
Our finding that esttaates of achievement rates and Judgsients of success 
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CITY M MEANS, STANDARD DEVIATiraiS AND MONTHLY RATE OF ACHIEVltffiNT O^R TWO 
TIME PERIODS, IN GRADE EQUIVAMNTS PGR TIffi GATlS-MacGINITIl 
READING TEST AVERAGED ACROSS COHORTS FOR STUDENTS IN 
Tlffi MATCHED, LOTGITUDmAL SmBlE 



Means and Standard Deviations 
in Grade-Equivalent Years 



Moathly Rate of 
Aehievement in Grade 
Equivalent Months^ 







I 


III 


III 


IV 


v 


Grade 


N 


fall 




Fall 


Fall to 


fall to 
Fall 


Third 


272 


2.23 
(1.04) 


3.29 
(1.42) 


2.78 
( .96)- 


1.5 


•6 


Fourth 


931 


2.65 
( .83) 


3.58 
(1.19) 


3.18 
( .96) 


1.3 


.5 


Fifth 


980 . 


3.26 
( .99) 


4,30 
(1.38) 


4,01 
(1.30) 


1.5 


.8 


Sixth 


316 


3.85 
(1.20) 


4.78 
(1.47) 


4.42 
(1.32) 


1.3 


.6 


Seventh 


128 


4.35 
(1.24) 


5,25 
(1.68) 


4.95 
(1.41) 


1,3 


.6 



To calculate the monthly rate of achievements the nraber of grade-equivalent 
motitha gained during each period of time (based on the means in Goltmns I^ 
Ilj and III) ii divided by the number of grade-equivalent months between 
test adminiitrat ions. There were seven grade-equivalent months between the 
fall and spring test aiteinistrationSi. so the fall*to*spring gain is divided 
by seven. By definition a grade -equivalent year is composed of the grade* 
equivalerit months; therefore, the annual fall-tp^fall gain is divided by ten, 



CITY M mmS IN GRADE EQUIVAIIMTS FOR THE GATES-MacGmiTIl FADING 
TEST FOR STTOENTS IN THE MATCIED LffllGlTUDINAL SAMPI£ " 
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were dramatically affected by the lingth of time used in the evaluation 
was consistent across all the compensator education programs we investi- 
gated* These resulta are iuimnariged in Table 2. Specifically, the 
inclusion of the suimer months in an evaluation often reduces considerably 
esttoates of achievement and hence monthly achievemenC rates (a comparison 
of the last two coli^ws of Table 2 demonstrates such a reduction in monthly 
rates of achievement)* In other words ^ achievement gains made during the 
school year were not sustained even until the next fall* These findings 
are baaed on analysis of matched j longitudinal samples containing over 
8,700 students and are unaffected by the grade levtl, the subject areas 
the program^ the itandardiEed achievement test or the metric i^sed in the 
analysis. 

The results of our new studies explain the conttadictions ©f our 
first Title I report* The state Title I evaluatlQns were baaed mainly 
on data from fall and spring administrations of achievement testi--and 
the programs appeared to be quite successful. Statewide testing programs 
measure annual achievement j the same as our fall'-td-^fall measures* When 
the annual achievament gain is used as the criterion for success ^ compen- 
satory education seems to be having little effect. VTith a few notable 
eKceptionSj annual gains are not greater than expected | ^hlch is roughly 
seven grade-equivalent months. 
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. Table 2 

AVERAGE MONTHLY BEADING ACHIEVElffiNT RATES BY GRADE 
ACROSS PROGR^S FOR TWO TDffi PERIODS 



Number of Number of 

Grade Programs .Students Fall-to-Sprlng rall-to-Fall 

1 I 665 0.6 0.4 

2 I 582 1.0 0.7 

3 2 1,053 1.1 0.7 

4 2 2j042 1.1 0.6 

5 2 1,812 1,3 0.7 ■ - 

6 2 1,044 1,0 0.7 

7 6 1,342 1.3 0.8 

8 2 171 . 1.4 1.0 
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Our findings have important topliGatlons for the InCerpretation of 
previous Title I evaluations. Since m it evaluations have prmarily 
used data from only fall and spring actolnistratlons of achievement testSp 
their results may be misleading* If Title 1 has as a goal a sustained 
increase in achievement, and I believe this is an agreed upon goalj it 
should be recogntged that there is virtually no information on this from 
previous studies. We therefore urge that Title I be reeoniidered In 
light of whether or not increases in achievement are sustained. 

Our findings also have mplications for future research. - Since the 
st™ner is clearly a critical period of timm^ there need to be further 
studies of compensatory education strategies which might affect the loss 
of achievement during the sunmer. To dates no such strategies have been 
identified, although a number of possibilities such as year-round schools 
exist. 

In conclusion J we urge that districts administer achievement tests 
minimally each fall and preferably each fall and spring. These data 
would provide the capability for estijiating the extent to which school* 
year gains are sustained through the following sunmier. Both fall and * 
spring tests have the added advantage of allowing a separation of 
school^year and smimer achievement* Although this information is not 
critical for estijaates of annual gains j it is a valuable source of infor* 
mation for studying the extent and causes of suroer losses. If 5 for 
eKample, the phenomenon is a function of the measures used, the standardise 
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achievement ttstSj one would want to change the measures* not the progr^. 
If it Is a result of instructional techniques that mitigate againat reten- 
tion 5 then the techniques should be changed. Since there are no simple 
solutions (for examples there is little research to support the notion 
that sumier school would alleviate the suroer losses), it Is important 
to be able to detemine why the losses occur in order to develop appro- 
priate remedies. 
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